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Quality Assurance and Public Accountability 


PURPOSE 

The purpose of this publication is to provide an overview of how quality assurance 
(QA) bodies are accountable to the public for how they accredit or quality assure’ 
effective institutional performance in the form of outcomes for students, often re- 
ferred to as student success or student outcomes. The project consists of two parts. 
Part | is a mapping of practices of QA bodies with respect to performance measures 
related to student outcomes. On the basis of the findings in Part |, Part Il presents 
case studies of practices for public accountability used by selected QA bodies. The 
project publication concludes with some general remarks and ideas for future con- 
sideration. 


The Council for Higher Education Accreditation International Quality Group (CHEA/ 
CIQG) thanks the four QA bodies — the Accrediting Commission of Career Schools 
and Colleges (ACCSC), the Malaysia Qualifications Agency (MQA), the Tertiary 
Education Quality and Standards Agency (TEQSA) and the WASC Senior College and 
University Commission (WSCUC) — whose practices are included in the report — for 
their support and contributions to the author in preparing the case studies. 


INTRODUCTION 

The world of higher education has been changing rapidly over the last 30 years with 
the move to mass education due to the needs of the knowledge economy for an ap- 
propriately skilled labour force and driven by continuous advances in technology. 


These changes have had an important impact on who is interested in higher educa- 
tion, how quality of higher education is defined and by whom. The changes are also 
putting pressure on higher education institutions and QA bodies to increase the 
transparency about how quality is assessed and what is required of higher educa- 
tion institutions or programs to be accredited so that useful information about edu- 
cational quality can be provided to the major stakeholders such as governments, 
students, parents, and the public, including the media. 


* The terms “accreditation/accredit” and “quality assurance” are used interchangeably in the report to refer to any quality assurance process, 
be it accreditation, review or audit 
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U.S-accreditors recognised by the U.S. Department of Education have, over the last 
10 years, gradually been required to increase information in the public domain 
about accreditation actions taken. The European Standards and Guidelines, ap- 
proved by the European education ministers and as revised in 2015, require the 
publication of full reports from QA agencies and all formal decisions taken on the 
basis of quality assurance reports, leading to both positive and negative outcomes. 


This project defines y 4 


public accountability as the 
responsibility of 
accreditation/QA bodies to 
provide reliable evidence of 
the effective performance of 
accredited or quality 
assured institutions and 
programs. 


ee 


Regardless of the increased degree of transparen- 
cy of the outcomes of quality assurance process- 
es, there is regularly public debate in many coun- 
tries about whether higher education institutions 
are preparing students adequately for employ- 
ment and are reflecting society’s expectations 
and needs. The emphasis on higher education 
being a public good has played an important role 
in repositioning higher education. This has been 
reflected in a demand for greater accountability 
for all public organisations, and there are specific 
issues for higher education regarding concerns 


about higher education access/participation, costs/debt, graduate employability/ 
unemployment, and social/economic impact. There is a growing desire to move 
beyond assessing quality to linking quality to relevance and resources. (Hazelkorn & 
Gibson, 20173, Eaton, 20172, Wilson, 2013%) 


These discussions and pressures provide the context for this CHEA/CIQG project 
Quality Assurance and Public Accountability. This project defines public account- 
ability as the responsibility of accreditation/QA bodies to provide reliable evidence 
of the effective performance of accredited or quality assured institutions and 
programs. The central feature of “performance” is what happens to students, i.e., 
completion, graduation, jobs and earning rates, and entry to graduate school or 
successful transfer. The information is to be available to all major stakeholders — 
students, institutions, programs, government, the public, and the media. 


The definition of accountability that is used aligns with typical definitions of “ac- 
countability” that generally focus on the idea of giving account of one’s actions 
to some other person or body’. Trow has identified five fundamental questions 
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with respect to accountability: who is to be held accountable, for what, to whom, 
through what means, and with what consequences’. These questions have guided 
the presentation of the case studies included in this publication. 


It should be mentioned that there are other means through which QA bodies are 
held accountable to stakeholders or take initiatives that provide an insight into the 
way they operate and the decisions they take. These measures include, e.g., regular 
external reviews conducted by professional associations or governments, govern- 
mental audits, thematic reviews of the main outcomes of their quality assurance 
activities, and databases containing current and past quality assurance outcomes. 
These measures of accountability are not included in this study. 


OVERVIEW OF QUALITY ASSURANCE BODIES PRACTICES 

The study includes 28 QA bodies across Africa, Austral-Asia, Central 

and LatinAmerica, Europe, the Middle East and North America (see (J) 
Annex A). The QA bodies have been selected in order to gauge how 

they are accountable to the public regarding their assessment of ef- 

fective institutional performance in the form of student outcomes. All the studied 
QA bodies have been in operation for a substantial period of time, i.e., they have 
performed at least one quality assurance cycle. The selection of the QA bodies has 
been informed by academic papers, QA body websites, and experts with knowledge 
of the particular QA bodies and their activities as they relate to public accountabil- 
ity. This approach does not mean that QA bodies not included in the study do or do 
not cover the performance measures included in the study in their accreditation/ 
quality assurance work and have good practices for public accountability. 


The project has primarily been conducted as desk-based research making use of the 
websites of the QA bodies involved and some clarification/follow-up directly with 
some QA bodies. The information that has been used to identify the accountability 
practices has been found in Standards & Criteria, Accreditation/Review/Audit Hand- 
books, Team/Reviewer Manuals, Guidelines for institutions, evidence/reporting 
templates, and Accreditation/Review reports/action letters available in the public 
domain. 


Some of the studied QA bodies only quality assure either institutions or programs; 
some work at both program and institutional levels. There are some QA bodies that 
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only include performance measures in one of 
the applied approaches. The mapping exercise 
showed that most of the 28 QA bodies studied 


Predominant performance 


require information on performance that in- measures related to 
cludes student outcomes at institutional and/or student outcomes are 
program level, depending on the type of quality attrition/retention, 
assurance approach they apply. For accreditation progression, completion, 
at institutional level, the performance measures graduation, and 


are either reported for the institution’s programs = employment. 


or generically at institutional level. a 7 


The predominant performance measures related 

to student outcomes covered by the QA bodies are attrition/retention, progression, 
completion, graduation, and employment (see table 1). The QA bodies’ approaches 
to considering these measures differ, however. Some QA bodies require institutions 
to report on specific measures, data, or indicators in a self-study, and they pro- 

vide definitions of these measures for institutions to use. Other QA bodies require 
institutions to identify and define the performance data/information they consider 
to be relevant in their particular context and report on these in a self-study. It is a 
common feature among the majority of the QA bodies that they require institutions 
to analyse and monitor their program or institutional performance data and be able 
to explain how they take action informed by their analysis of the data. 


Institutional performance is in most cases assessed in the accreditation or quality 
assurance process in the context of an institution’s or program’s mission and vision, 
i.e., a fitness-for-purpose approach. These assessments lead to a holistic consider- 
ation by reviewers and recommendations to the QA bodies as to whether the insti- 
tutions or programs meet the QA body’s standards, criteria, or requirements. There 
are very few examples where QA bodies have defined threshold or baseline perfor- 
mance outcomes for successful accreditation, and this information is available in 
the public domain. Only one QA body has been identified that currently publishes 
threshold/ benchmark statements about the expected institutional performance 
with respect to student outcomes. In the cases where QA bodies provide informa- 
tion about institutions’ student outcomes to the public, it is provided in accredita- 
tion reports or action letters to institutions. 
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It is worth noting that the mapping also showed that half of the studied QA bodies 
assess information about institutional or programmatic finances, i.e., information 
about financial viability or sustainability. Although these measures do not reflect 
student achievement, the financial stability of an institution is an important indica- 
tor of the institution’s ability to perform effectively and achieve its defined student 
outcomes. 


Table 1: Performance Measures Covered by Studied QA Bodies 


Performance Measure HKole-] Mell] me) m2.) 
Attrition/Retention 
Progression 
Completion 
Graduation 


14 


RiP 
ON] 


Employment 

Credit transfer 

Continuation of studies 

Debt 

Loan default 5 


Earning rates/information 


Note: If a QA body conducts more than one type of accreditation covering a performance measure, it 
is only counted once. 


[Employment 
[Credittransfer 


In summary, the mapping shows that data about student outcomes feature in both 
institutional and program quality assurance and that the most frequently collected 
data are used to assess if students complete their studies and to determine their 
employment situation after graduation. The responsibility for identifying student 
outcomes data varies and either rests with the QA body or is left to the institution 
itself. It is not common practice for QA bodies to define particular thresholds for ac- 
ceptable performance. Nevertheless, there is a general expectation from QA bodies 
that institutions collect, analyse and act on student outcomes data to maintain and 
improve quality. 
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CASE STUDIES 
Based on the mapping, four QA bodies have been identified to be used for case 
studies. 


The following section presents four different approaches or tools used by QA bodies 
to determine an institution’s or a program’s performance, how the QA body’s ex- 
pectations of that performance is presented to stakeholders and what recommen- 
dations, if any, are made by the QA body for institutions to improve performance. 
The case studies are intended to provide examples/inspiration and/or be the start- 
ing point for considerations and discussions about ways in which transparency of 
public accountability of QA bodies’ assessment of student outcomes can be devel- 
oped or enhanced. The case studies are not intended to provide a full presentation 
of the chosen QA bodies’ quality assurance activities. They focus on the approach 
to public accountability. 


Each case study is divided into three sections: 
1. Brief introduction of the QA body to provide the context of the presentation 
of the tool for assessment of performance 
2. Presentation of the tool to assess performance 
3. Presentation of how outcomes of the assessment of the performance are 
made transparent to stakeholders 


Case study A: Accrediting Commission of Career Schools and Colleges 


Introduction 

The Accrediting Commission of Career Schools and Colleges (ACCSC) is recognised 
by the United States Department of Education (USDE) as the designated institution- 
al accrediting body working with around 700 postsecondary, trade, and technical 
schools. 


ACCSC’s scope of (USDE) recognition covers the accreditation of both non-degree- 
granting and degree-granting institutions, including those granting associate, bac- 

calaureate, and master’s degrees. To be eligible for ACCSC accreditation, a school’s 
educational objectives must be career-oriented and aimed at providing graduates 

with the necessary competencies, skills, and level of education for employment in 

their fields of study. 
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ACCSC’s accreditation processes focus on inputs, resources, and processes, i.e., 
how the school operates. All of these areas are evaluated within the context of the 
school’s stated mission and its demonstrated achievements. The accreditation pro- 
cesses also promote institutional self-evaluation and improvement and provide for 
public accountability within a peer-review framework. 


ACCSC accreditation is based upon criteria established via ACCSC’s Standards of Ac- 
creditation. The standards emphasise educational quality by focusing on outcomes. 
The standards include a requirement for institutions to report student achievement 
outcome data such as retention, graduation, and employment data, as well as licen- 
sure/certification exam pass rates against established benchmarks. 


The ACCSC has Wea, 


benchmark rates of student 
graduation and graduate 
employment for its accredited 


Performance Assessment Tool: Benchmarks for 
Student Outcomes 

The ACCSC has established benchmark rates of 
student graduation and graduate employment 


for its accredited schools from information col- schools from information 
lected in its ongoing accreditation processes collected in its ongoing 
and the institutions’ annual reporting to ACCSC. let he itn sai 
Until 2016, ACCSC recalculated student achieve- reporting to ACCSC. 

ment benchmark rates every three years using 

a rolling average. In 2016, ACCSC conducted an a 


11-year longitudinal study of the graduation and 

employment rates reported from 2005 to 2015 in the Annual Reports in order to 
discern the stability, predictability, and reliability of the rates used in establishing 
ACCSC’s benchmark rates. The longitudinal study applied the same formula to the 
11-year data set as ACCSC had applied to the three-year data sets previously used. 
Overall, ACCSC found that the data collected and the resulting benchmarking estab- 
lished from that data were both stable and reliable, and there was little variation 
from ACCSC’s existing benchmarks compared to the results of the 11-year longitudi- 
nal study. 


ACCSC provides graduation and employment charts for institutions to complete as 
part of the accreditation processes. The charts include the required information, as 
well as the data definitions. Schools are expected to set goals to exceed not only 
the benchmark rates but also the average rates of graduation and employment®. 
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Table 2: ACCSC Student Achievement Rates in Effect for all Graduation and Em- 
ployment Charts That Use a Report Date of July 1, 2016 and Later 


Established Benchmark Graduation Rates 


Sources: ACCSC Standards of Accreditation and Bylaws - 070117 final 


: E lished Bench- 
Program Length in | Average Rates of Standard steplisned tes 
E es mark Graduation 
Months Graduation Deviation 
Rates 
1-3 92% 8% 84% 
4-6 84% 11% 73% 
7-9 72% 12% 60% 
10-12 69% 14% 55% 
13-15 64% 14% 50% 
16-18 62% 15% 47% 
19-23 61% 18% 43% 
24+ 53% 13% 40% 


ACCSC also requires a 70% licensure pass rate on certification exams when they are 
required for a training program by law or regulation for employment.” 


ACCSC reviews student achievement for each program offered at an institution at 
least annually via the submission of an institutional Annual Report and considers 
not only the rates at which students graduate from a training program, attain em- 
ployment in a training-related field, and pass licensure/certification exams required 
for employment, but also other factors that are reasonably related to student 
achievement as a measure of educational quality and institutional effectiveness. 


For any program that has a graduation, employment, or licensure/certification 
exam pass rate that is lower than ACCSC’s established benchmark rates, a school is 
given an opportunity to demonstrate, with supporting documentation, the success- 
ful achievement of its students in that program by providing other reliable indica- 
tors of successful student learning and by showing that factors such as economic 
conditions, state and national trends, location, student population served, length of 
program, students who withdraw from training but still obtain employment, state 
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requirements, or other external or mitigating factors reasonably related to student 
achievement are adversely impacting the school’s ability to meet the established 
benchmark rates. 


The ACCSC Commission considers the student outcomes data as part of its holistic 
assessment of an institution. As part of its efforts to recognise high performing insti- 
tutions, ACCSC offers an additional year of accreditation, i.e., a six-year reaccredita- 
tion period instead of a five-year reaccreditation period for those schools that suc- 
cessfully complete the accreditation process without any compliance, regulatory, or 
financial findings AND demonstrate that the majority (50% of more) of the programs 
offered at the school report a graduation and employment rate at AVERAGE or better 
(not benchmark). 


Any program that has a graduation, employment, or licensure/certification pass 
rate that falls below the ACCSC’s established benchmark rate will be subject to 
heightened monitoring or reporting of student achievement outcomes as directed 
by ACCSC or to other action as deemed appropriate. 
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Table 3: Examples of ACCSC Action 


Requirements 


Heightened monitoring At a minimum, detailed annual review of a pro- 
gram’s graduation and/or employment rate(s). This 
may require the submission of additional student 
achievement outcomes information as appropriate 
(e.g., plans for improvement; an updated Graduation 
and Employment Chart; current program retention, 
graduation, employment and/or licensure/certifica- 
tion pass rate(s); supporting documentation; etc.). 


Reporting The submission of student achievement outcomes 
data (e.g., student program retention, graduation 
rates, and employment rates); pass rates on licen- 
sure/certification examinations and examinations 
required to be taken as a condition of employment 
(i.e., local, state, and federal); admissions criteria 
studies; institutional operations and improvement 
planning; or any other information that the Com- 
mission determines necessary to make a judgment 
regarding the successful achievement of students. 


Programmatic action Require an on-site evaluation; temporarily require a 
school to cease enrollment in a program; suspend or 
revoke program, degree-granting, or distance educa- 
tion approval. 


Institutional action Issue a Warning or Probation Order when the Com- 
mission has determined that the school has not 
demonstrated acceptable student achievement 
either through its student learning assessment ef- 
forts; graduation, employment attainment, and/or 
licensure/certification exam pass rate(s); or a combi- 


nation thereof. 
Source: ACCSC Standards of Accreditation and Bylaws - 070117 final 
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Transparency of Assessment of Performance 

The institution’s student achievement data is provided in the institutional Self-Eval- 
uation Report and summarised in the Team Summary Report (TSR) issued following 
an on-site visit to an institution. 


If the reported graduation, licensure, or employment rate for any program is below 
benchmark, the TSR will include a “Finding” that the school must respond to. That 
response is provided to the ACCSC Commission for consideration. Typically, the 
school will submit a response describing its strategies and initiatives that have been 
implemented to help improve the reported rates of graduation and employment, as 
well as more recent Graduation and Employment Chart to show how the rates have 
trended since the visit. 


The institutional response is also provided to the ACCSC Commission to inform its 
decision making. If the rates of graduation and employment continue to fall below 
benchmark, the Commission’s action letter, i.e., the reporting of its decision to the 
institution, will include particular requirements for the school address. If the rates 
fall significantly below benchmark, or if the rates fall continuously below bench- 
mark (reporting data to commission for an extended period of time), this could 
result in a programmatic action or institutional action. 


The ACCSC Commission may order a school to submit a report on its compliance 
with accreditation requirements during the course of routine accreditation reviews 
or at any time the Commission believes that monitoring of compliance with an ac- 
creditation requirement is warranted. 


In cases where the Commission has significant concerns regarding a school’s com- 
pliance with one or more accrediting standards or other requirements or has de- 
termined that a school is out of compliance with one or more standards, the Com- 
mission may, at its discretion, place a school on Probation. As part of the Probation 
Order, the Commission may direct the school to show cause as to why the school’s 
accreditation should not be withdrawn, and it will be required to demonstrate cor- 
rective action and compliance with accrediting standards. 


ACCSC actions are summarised and available on the ACCSC website, while full 
Probation and Adverse Action letters, e.g., regarding denial or withdrawal of ac- 
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creditation, are published on the ACCSC website. Under ACCSC’s Rules, on a case- 
by-case basis, the Accrediting Commission may also elect to publish Warning letters 
on the website. ACCSC’s Standards of Accreditation also require a school subject to 
a Probation Order to inform current and prospective students that the school has 
been placed on Probation and that additional information regarding that action can 
be obtained from the Commission’s website. When student achievement results 
affect the Commission’s decision to place an institution on probation, a comprehen- 
sive analysis of the student achievement data is included in the action letter to the 
institution. 


ACCSC offers a range of special workshops and webinars that focus on ro) 
best practices of high-performing institutions and publishes a diverse ar- Po 
ray of materials designed to support institution not only to demonstrate S24 
compliance but also to support the schools’ ability to ensure students 

receive high-quality education. This information is published on the ACCSC website, 
where it is available not only to member institutions but to the public at large. 


Case study B: Malaysian Qualifications Agency, Malaysia 


Introduction 

The Malaysian Qualifications Agency (MQA) is the national quality assurance agen- 
cy. It was established in 2007 by the government under the Malaysian Qualifications 
Agency Act 2007. MQA is the guardian of the Malaysian Qualifications Framework 
(MQF) and is responsible for overseeing quality assurance practices and accredita- 
tion of national higher education at both institutional and program level. This role 
involves the development of institutional and program standards and criteria and 
provisional accreditation of new programs, followed by full accreditation of both 
public and private higher education providers®. All higher education institutions are 
subject to maintenance audits to determine the continuation of program accredi- 
tation status and to periodically review of the institution’s academic activities and 
performance. The audit is a tool for the Ministry of Higher Education to facilitate 
the monitoring of the quality of HEIs’. The MQA program standards include a re- 
quirement for institutions to provide information on student attrition, progression, 
completion, and credit transfer. The institutional standards include a requirement 
for information on attrition and transfer of credit. 
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Performance Assessment Tool: Discipline Ratings 

In addition to its accreditation activities, MQA in 2009 and 2011 conducted a 
Discipline-Based Rating process (D-SETARA) that covered schools or faculties where 
particular disciplines were offered. The purpose of D-SETARA was to assess the 
quality of teaching and learning at level 6 (undergraduate) of the MQF in universi- 
ties and university colleges. The intention behind the approach was to benchmark 
disciplines against a set of quality standards for the purpose of improving the 
institution and the discipline. The approach was not intended to lead to competi- 
tion between institutions. The D-SETARA is complementary to I-SETARA, i.e., ratings 
at institutional level that have been carried out for public universities and university 
colleges from 2007” by the Ministry of Higher Education. 


The D-SETARA instrument was developed by subcommittees established by MQA, 
and the methodology was based on the I-SETARA instruments, in order to ensure 
comparability with the institutional ratings. The D-SETARA evaluation of teaching 
and learning was divided into Input-Process-Output domains. Each domain con- 
sisted of a set of criteria that were divided into a number of indicators. Each domain 
had been allocated a weight which differed based on the type of discipline. For 
example, in 2011 the weight was 20:40:40 for the three domains for the Engineer- 
ing and the Medicine, Dentistry and Pharmacy disciplines; 30:40:30 for Health Sci- 
ences, and 30:35:35 for Hospitability and Tourism. The varying weighting system of 
domains, criteria, and indicators was justified by differing discipline-specific needs. 


Data for the indicators were collected from various sources, which included institu- 
tional data (through self-assessment supported by documentary evidence), em- 
ployer surveys, tracer studies, and accreditation data. Various steps were taken to 
ensure data integrity from each of the sources. For the purpose of data collection, a 
template was prepared for the higher education institutions to use. The data sub- 
mitted by the institutions were compiled and then verified by the subcommittees. 
Two verification meetings were conducted. The verification exercises covered the 
process of instrument development data collection, entry and analysis; missing data 
analysis and treatment; and the results. Subsequently, data analysis was undertak- 
en which ultimately resulted in a tiering of the higher education institutions in each 
discipline. 


**1n 2007 I-SETARA only included public universities. 
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The “Outcome” domain included two student performance measures, i.e., criterion 
1: Graduate Marketability, which was divided into employment rate and average 
monthly start pay, and criterion 5: Student Performance, which covered the rate of 
students completing their studies within the stipulated time. 


Table 4: Engineering Output Domain D-SETARA 2011 


OUTPUT: QUALITY OF GRADUATES AND GRADUATE SATISFACTION 


Criteria 


Graduate 
Marketability 


Graduate 
Satisfaction 


3 Employers’ 
: Satisfaction 


Generic 
Student 
Attributes 


Student 
Performance 


Competitions 
and Awards 


Sub- 
Weight 


Indicators Weight 


: Source of Data 
Unit Benchmark | Full 
Data Marks 


Tracer Study 
Q40 and 54 


Flexible Tracer Study 044 
Score 5 . Tracer Study 
Q20, 21, 22, 23, 
24, 25 and 26 (b) 


Score | 5 
Employer 
Satisfaction 


Survey 


Employment rate 
(six months after 
completion of 
studies or continuing 
studies) 


Average monthly 
starting pay of 
graduates who are 
employed. 


Average satisfaction 
with the Higher 
Education Provider (HEP) 
of the graduates. 


100.00 


Average satisfaction 
with the graduates of 
the HEP 


Intention to employ 
from the HEP Segre 
Employer 
Satisfaction 
Survey/Tracer 
Study Q27 


They seven generic 
student attributes 


yl 
° 


Percentage of students 
completing within 
the stipulated time 


a 
fo) 


Number of participation 
by students in 
international or national 
engineering-oriented 
competition per program 


Institutional 
Data 


Number of achievement 
by students in 
international or national 
engineering-oriented 
competition per program 


Number 


Total Marks for Quality of Graduates and Graduate Satisfaction 


Source: MQA website 
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Transparency of Assessment of Performance 

The D-SETARA exercise resulted in a six-level rating of the participating higher 
education institutions in each discipline. The six tiers were: Outstanding, Excellent, 
Very Good, Good, Satisfactory and Weak. The basis for the rating was the total 
scores and cut-off values determined for each of the tiers. Only the overall rating of 
an institution was published. The marks achieved for each indicator; the minimum, 
maximum and mean marks achieved for each criterion; and the overall marks of 

an institution were communicated to the respective institutions only. The overall 
rating, in terms of the institutional tier rating of the discipline, was announced ina 
press conference hosted by the Minister for Higher Education, and announcements 
were made in the major newspapers. The higher education institutions received 
their results by letter. There was no link between MQA’s program accreditation pro- 
cesses and D-SETARA as the outcomes of the D-SETARA instrument were intended 
to be used by higher education institutions for improvement purposes. 


Case study C: Tertiary Education Quality and Standards Agency, Australia 


Introduction 

The Tertiary Quality and Standards Agency (TEQSA) is Australia’s national qual- 

ity assurance and regulatory agency for higher education. TEQSA was established 

in 2011 by the Commonwealth government through an Act of Parliament — the 
TEQSA Act. All providers, universities and non-universities, that offer higher educa- 
tion qualifications in or from Australia must be registered by TEQSA and undergo 
re-registration at least every seven years. Higher education providers that do not 
have self-accrediting authority must also have their courses of study accredited and 
reaccredited by TEQSA at least every seven years. TEQSA regulates higher education 
through a National Standards Framework that is developed by an expert Standards 
Panel, a legislative body included in the TEQSA Act. The threshold standards require 
that institutions review and monitor student success and provide their analyses of 
student progression rates, attrition rates, completion times, and rates to TEQSA. 


The TEQSA Act specifies that TEQSA’s regulatory approach is to be underpinned by 
three regulatory principles: risk, necessity, and proportionality. TEQSA considers risk 
to primarily mean the risk of current or future provider non-compliance with the 
threshold standards and associated risks to quality, to students, and to the reputa- 
tion of the Australian higher education sector. TEQSA applies a systematic, struc- 


Quality Assurance and Public Accountability Page 15 


tured, and consistent approach to assessing compliance risk across all providers, us- 
ing a standard set of risk indicators corresponding to primary areas of institutional 
practice and performance. The risk indicators are contained in a Risk Assessment 
Framework and are applied in annual institutional risk assessments”. 


Performance Assessment Tool: Risk Assessment 
TEQSA focuses on four main areas in its assessments of risk. They are: 
1. Provider regulatory history and standing 
2. Students (load, experience, and outcomes) 
3. Academic staff profile 
4. Financial viability and sustainability 


The TEQSA Risk Framework includes 11 indicators, four of which are directly related 
to or cover aspects related to student outcomes. 


Table 5: TEQSA Risk Indicators 
Risk Indicators a Outcome Measures 


Student load 


Attrition rate Xx Es transfer to other 
institutions 


Progressrate IX 
Completions 
Graduate satisfaction SS 


x 

[Student-to-staffratio | 
[Academic staffon casual work contracts | 
[Financial viability | 
[Financial sustainability | 


TEQSA can cover other identified risks, e.g., provider specific risks. 
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A key aspect of TEQSA’s approach to assessing and arriving at a rating of an indica- 
tor is to consider a provider’s specific circumstances, which will supply the context 
for understanding any potential risks that the indicator is designed to capture. 
TEQSA also considers the calculated quantitative value of the indicator, based on its 
published technical definition, and with reference to ‘risk thresholds’. Risk thresh- 
olds provide guidance to TEQSA’s assessment of areas in which potential risks may 
be present. Considered together with the individual circumstances and context of 
the provider, TEQSA derives a risk rating (represented by traffic lights: green, yellow, 
and red). The assessment of individual indicators is then considered holistically to 
inform an overall risk evaluation of a provider. 


The risk assessment is shared with each provider annually, except for newly regis- 
tered providers that may have insufficient data to conduct a risk assessment. Provid- 
ers are invited to respond to their risk assessment, which may lead to adjustments 


in the risk assessment. 


If significant risks are identified, the provider is invited to discuss the risk assess- 
ment and provide any broader context and information on its strategies and any risk 
controls it has in place. The finalised risk assessment is used to inform the scope 
of scheduled assessment processes, e.g., renewal of registration application pro- 


cesses". 


A final risk assessment will typically identify action in line with the following: 


No action 


nXexeelaavanteyarere areal 


If no significant risks are identified overall or risks are al- 
ready known to TEQSA with a response already in place 
(such as additional reporting requirements), then TEQSA 
will not take any action in response to the risk assess- 
ment. The risk assessment will continue to be updated 
annually. 


TEQSA may recommend that the provider closely 
monitor identified risks and/or put in place appropriate 
controls or improvement strategies. A recommendation 
arising from a risk assessment does not constitute a 
formal condition on registration. 
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TEQSA may identify risks that require further consider- 
ation by TEQSA. In such cases, TEQSA may seek addi- 
tional information from the provider so that TEQSA may 
determine if further action is necessary. Requests for 
information may also be used to monitor identified risks 
between risk assessment cycles. 


If TEQSA identifies significant risks, it may determine 
that regulatory action is necessary outside a scheduled 
Nau) Elceeelealelal (729, | assessment process. This may include, for example, 
@leyeaye) yaa <sissspelseye | undertaking a compliance assessment to satisfy TEQSA 
foyarero)aelintolats}) that the provider continues to comply with the Thresh- 
old Standards or imposing formal conditions on registra- 
tion. 


Cele LUL=s] mace) au layxe)aaat-lateyal 


If the provider has a scheduled assessment process 
To be considered in (e.g., re-registration), TEQSA may indicate that risks 
Yel sl=re le} (Yeu enlplaee) identified in the risk assessment will be considered fur- 
process ther during that process rather than identify additional 


action at that time. 
Source: TEQSA Risk Assessment Framework v2 


In early 2018, TEQSA instituted a policy of meeting with all high-risk providers to 
discuss their performance. TEQSA has plans in the longer term to revise its Provider 
Engagement approach to include a visit to all providers on an annual basis. 


Transparency of Assessment of Performance 

The Risk Assessment Framework that includes the risk indicators is available to the 
public on the TEQSA website. The risk assessment of an individual provider is con- 
fidential between TEQSA and the provider. The risk thresholds are not published or 
shared with providers. The reason for the confidentiality of the risk thresholds are: 


e Arisk assessment does not, on its own, result in definitive conclusions in 
relation to compliance with the Threshold Standards. 

e = Ill-informed commentaries surrounding the risk thresholds could be very 
damaging to the reputational and commercial standing of providers and the 
sector. 
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e Qualitative information, including contextual and control information, is 
an important input to TEQSA’s rating of a risk indicator, which goes beyond 
quantitative risk thresholds. 

e There is a high risk of third parties misusing the risk thresholds in trying to 
replicate risk assessments, e.g., to rank providers.” 


In registration and re-registration processes where TEQSA has identified potential 
risks of a provider not being in substantial compliance with the threshold standards, 
TEQSA’s decision will include conditions as to what action the provider needs to 
take to improve the areas of non-compliance. The decision also includes an expla- 
nation why the decision has been made, a decision that reflects the three regula- 
tory principles risk, necessity, and proportionality. All conditions are publicly avail- 
able on the National Register, which is a public database that TEQSA maintains on 
its website. 


In October 2018, TEQSA published a report titled ‘Assessment Insights’, which 
analyses and presents the relationship between risk assessments and regulatory 
outcomes generically rather than on a provider basis. 


Case study D: WASC Senior College and University Commission, California USA 


Introduction 

The WASC Senior College and University Commission (WSCUC) is one of seven 
regional accrediting agencies in the United States. Regional accreditation serves to 
assure the educational community, parents, students, employers, policymakers, and 
the public that an accredited institution has met certain standards of quality and ef- 
fectiveness. Regional accreditation is voluntary and non-governmental. The institu- 
tions accredited by the regional accreditors, including WSCUC, comprise the region- 
al accreditors’ membership. Each regional accreditor has its own membership. 


WSCUC serves a diverse group of public and private higher education institutions 
throughout California, Hawaii, and the Pacific, as well as a limited number of insti- 
tutions outside the United States. The accreditation process is conducted based on 
standards agreed to by the WSCUC members, and its accreditation activities seek to 
ensure that the standards are met, as well as to encourage continuous institutional 
improvement. 
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Every accredited institution is subject to periodic 
review and to conditions, as determined by the 
Commission. A member institution has to file an 


Annual Report, provide information for a Mid- WSCUC has developed the 
Cycle Review, and undergo a comprehensive so-called Graduation Rate 
accreditation review every six to ten years. Dashboard that measures 


undergraduate retention 


The WSCUC accreditation standards require and graduation rates. 


that institutions provide disaggregated data on 

students’ progress toward timely completion of 

their degrees and on retention and graduation i a 
rates for the programs delivered by the institu- 

tion. WSCUC has developed the so-called Graduation Rate Dashboard (GRD) that 
measures undergraduate retention and graduation rates." 


Performance Assessment Tool: Graduation Rate Dashboard 

The GRD is included as one element of a comprehensive analysis of student suc- 
cess conducted during the WSCUC institutional accreditation review process. The 
GRD was developed to serve as a more inclusive method to identify the enrolment, 
retention, and graduation patterns of all undergraduate students. The primary 
benefit of this methodology is that it accounts for all graduates regardless of how 
they matriculate (first-time or transfer, lower or upper division) or enrol (part-time, 
full-time, swirling), or what programs they pursue. Unique to the GRD is that it pro- 
vides insight into the unit accumulation, redemption, and abandonment patterns of 
undergraduate students at the institutional level. 


Each year, as part of an institution’s annual reporting to WSCUC, all eligible, 
candidate, and accredited WSCUC institutions are required to provide six data 
points used in the GRD (see below). Other data elements, including graduation rate 
data, are collected from the National Center for Education Statistics (NCES). Student 
loan cohort default rates from the US Department of Education Federal Student Aid 
office are also included in the comparative tool. 
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The six GRD data points 


Unduplicated headcount of undergraduate degree-seeking students 


Total number of successfully completed graduate units awarded by the institution 


Undergraduate degree recipient headcount 


Cumulative total of successfully completed undergraduate units awarded by 
the institution to graduating students 


Non-continuing student headcount 


Total institutional units of non-continuing students 


The six data points generate two completion measures, the Unit Redemption 

Rate (URR) and the Absolute Graduation Rate (AGR). The unit redemption rate 
(URR) is the proportion of units granted by an institution that are eventually “re- 
deemed” for a degree from that institution. The absolute graduation rate (AGR) is 
the proportion of students entering an institution who graduate from that institu- 
tion, regardless of how long it takes them. The unit redemption rate is an alterna- 
tive way to measure completion, somewhat different from the standard graduation 
rate, as it counts units rather than students. The URR counts units that all students, 
full time and part time, first-time and transfer, take and redeem. Ideally, every unit 
granted by an institution ultimately results in a degree, but no institution actually 
achieves this ideal, given that students who drop out never “redeem” the units 
they take while enrolled. Conversely, some students may take more credits than are 
required for a degree. Nonetheless, the URR is typically below 100 percent. 


By adding one additional data point (the average number of units taken by students 
who leave the institution), the URR is converted into a graduation measure, the 
absolute graduation rate, which estimates the proportion of students entering a 
college or university (whether first-time or transfer) who eventually graduate. 


Both the unit redemption rate and the absolute graduation rate are influenced by 
significant or sustained enrolment fluctuations. Specifically, a decline in enrolment 
will increase the URR and the AGR, and an increase in enrolment causes a drop in 


Quality Assurance and Public Accountability Page 21 


both measures. Due to such fluctuations, it is possible to have outcomes greater 
than 100%. For this reason, it is important to take enrolment patterns into consider- 
ation when using these measures. Both the URR and AGR measures are comparable 
across all institutions. 


In summary the GRD: 
e Builds on six data points 
e Covers eight years of trends 
e Results in two completion measures 


Transparency of Assessment of Performance 

GRD results are being considered as one measure of student success in WSCUC re- 
views. WSCUC accreditation includes a review of the Integrated Postsecondary Edu- 
cation Data System (IPEDS) cohort default rates, publicly posted information on 
student achievement, and any other measures provided by the institutions as part 
of accreditation, reaccreditation, and mid-cycle review. WSCUC does not intend 
that any accreditation decisions will be made based solely on any one measure, 
including the GRD measures. The Commission is committed to the use of multiple 
measures of retention and graduation to gauge institutional effectiveness and stu- 
dent success.“ 


There are no specific graduation rate thresholds that institutions are required to 
meet to have a successful (re)accreditation outcome. Rather, institutions are asked 
to “engage” with the GRD, determining whether it adds to their understanding 

of student success or introduces new questions or data queries that are of value. 
Peer evaluator team reports comment on retention and graduation rate data, on 
trends in these data, and on institutions’ approaches to monitoring and evaluating 
retention and graduation data. WSCUC has developed a comparability tool which is 
available on the WSCUC website as a resource for institutions and peer evaluators 
to better understand and improve graduation rates. 


WSCUC provides instructions on its website on the use of the GRD in evaluator 
team training and regularly conducts workshops at relevant conferences. In addi- 
tion, WSCUC is currently exploring the use of benchmarking tools that would allow 
for estimating predicted retention and graduation rates based on certain character- 
istics of an institution and its students using IPEDS and GRD data. 


Council for Higher Education Accreditation/CHEA International Quality Group Page 22 


CONCLUDING REMARKS 

As mentioned in the introduction, the level of transparency of accreditation out- 
comes, positive as well as negative outcomes, has gradually increased, and in many 
parts of the world, it has moved beyond reporting only the decision, i.e., whether 
accreditation or a QA process had been successful or not, to publication of, e.g., full 
reports, Summary reports, or action letters. This development has been driven by 
an increasing demand from stakeholders for a higher degree of insight into institu- 
tional performance and how it affects student outcomes. The study suggests that 
the information listed below could be relevant for QA bodies to take into account 
when communicating with stakeholders to strengthen the communication about 
student outcomes. 


Generic definition(s) of student outcomes 

Performance measures used to assess student outcomes 

Descriptions of data/data points/indicators used to assess student outcomes 
Thresholds used, if any, to determine acceptable levels of student outcomes 
Actions taken when student outcomes do not meet expectations 

Trends in levels of student outcomes consolidated across institutions accred- 
ited 

7. Link to publications of institutional student outcomes. 


Au PWN 


If a QA body were to provide information to stakeholders about student outcomes 
along the lines suggested above, it would have to consider how the information 
could best be provided in line with its communication strategy. The template for 
public reporting of student outcome measures below is included as an example 
only. 
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Template for Public Reporting of Student Outcome Measures 


Reporting 
Generic definition(s) of student 
success 


Performance measures used to 
assess student outcomes 


Data/data points/indicators used 
to assess the student outcomes 


Thresholds used, if any, to deter- 
mine acceptable levels of stu- 
dent success 


Actions taken when student out- 
comes do not meet expectations 


Possible content 


Present to stakeholders how student success is 
defined in terms of outcomes in the accredita- 
tion/quality assurance processes. 


Present the specific measures, such as reten- 
tion, attrition, graduation, used in the review 
process and how these measures are defined 
to guide institutions in their self- review or 
internal quality assurance processes and to 
guide teams in the external review process. 


Provide examples of the data that institutions 
are expected to provide as evidence of their 
student outcomes and examples of how the 
data are monitored and acted upon by the 
institutions. 


Include information about the thresholds used 
by the QA body to determine if the outcomes 
are satisfactory from an accreditation/qual- 
ity assurance perspective or whether action is 
required to improve performance or further 
explanations are required to understand the 
reasons for the outcomes. 


For institutions that do not meet thresholds, 
provide information on the actions the QA 
body requires them to take to improve per- 
formance. This could include information 
about how the performance was eventually 
improved. 
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Trends in levels of student 
outcomes consolidated across 
institutions accredited 


Link to publications of institu- 
tional student outcomes 


Provide analysis of the how student outcomes 
have developed across all accredited or quality 
assured institutions over time and what trends 
have been identified to require particular at- 
tention for improvement or what areas have 


been recorded as particular strengths 


Provide a link on the QA body website that 
gives stakeholders access to the publications, 
such as action letters, accreditation reports, 
and decision documents, that include informa- 
tion about institutional student outcomes. 


The case studies present four very different tools used by QA bodies as means of 
public accountability. The mapping of the studied QA agencies shows that, in most 
cases, student outcomes related to completion of studies and employment are in- 
cluded in the quality assurance processes, but specific definitions of thresholds for 
acceptable or low performance were rarely provided to accredited institutions and/ 
or to the public. Also, the study identified no commonalities in the way that the 
performance of student outcomes at accredited or quality assured institutions are 


presented to the public. 


Nevertheless, the study shows what type of information is used by a large number 
of QA bodies to assess the student outcomes at institutional or program level, and 
it represents what might be useful for the stakeholders to access. 


IDEAS FOR FUTURE CONSIDERATION 


As the discussion about accountability and institutional performance is 
likely to continue, this study might provide inspiration for QA bodies in (-) 
consultation with their main stakeholders to discuss: = 
1. pros and cons for a higher degree of transparency with respect to 
expected and/or acceptable levels of student outcomes 
2. identification of relevant publicly accessible information about acceptable 
levels of student outcomes performance 
3. effective approaches for making the information easily accessible to various 
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stakeholders in higher education, e.g., through the use of an information 
template 

4. how QA bodies can encourage institutions to improve performance to 
generate strong student outcomes 


These recommendations are made in the context that there is not one size that fits 
all for how to provide information about acceptable levels for student outcomes 
and how this information is provided to the public. 


Council for Higher Education Accreditation/CHEA International Quality Group Page 26 


ANNEX A: List of Quality Assurance Bodies 


| | Abbreviation | Quality Assurance Body Country/Territory 


Niall -—ieckaeaiaed Mia 
Technology 
see | 
Schools and Programs 
iA aeeeiiaad 
tion & Training 
CLs lemme | 
and Junior Colleges 
= ice 
Schools and Colleges 
CL licegerommerenen |e 
in Nursing 
eT 
ates 
iil aaa 


i. 4 CNA Colombia | National Council of Accreditation 


DEAC Distance Education Accrediting Commis- USA 
sion 
E FINEEC __| Finnish Education Evaluation Centre 
| HEQC Higher Education Quality Committee South Africa 
HKCAAVQ | Hong Kong Council for Accreditation of Hong Kong 
Academic and Vocational Qualifications 
| | HLC | Higher Learning Commission 


KHDA The Knowledge and Human Development Dubai, United 
Authority Arab Emirates 
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ANNEX A: List of Quality Assurance Bodies 


NECHE New England Commission on Higher USA 
Education 


| | MQA __| Malaysian Qualifications Agency 


MSCHE Middle States Commission on Higher Edu- USA 
cation 

NVAO Accreditation Organisation of the Nether- | Flanders and the 
lands and Flanders Netherlands 

NWCCU The Northwest Commission on Colleges 
eae Universities 


QAA |QAA England and Northern Ireland and Northern Ireland United Kingdom 


— QAA Scotland 
| | aq | Quality and Qualifications Ireland 


ll = 
Schools Commission on Colleges 

| | SINEAS [National Accreditation Council |__—CostaRica__| 

| | TEAC __| Teacher Education Accreditation Council |___— USA 


TEQSA Tertiary Education Quality and Standards Australia 
Agency 

WSCUC WASC Senior College and University Com- USA 
mission 
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